Media composition system with remote connection

ABSTRACT

Computer-based methods for time-based media composition of programs that include secondary material. The methods involve searching for secondary media available from secondary material providers over a wide-area network, selecting secondary material, receiving metadata pertaining to the selected secondary material, inserting a representation of the selected secondary media into a media composition, playing back a sequence of the media composition that includes the secondary media, wherein playback of the secondary media involves playing an evaluation version of the secondary media streamed to a media composition system over the wide area network, and displaying metadata pertaining to the selected secondary material. The media composition system facilitates purchasing and transfer, and re-linking of the secondary media for inclusion within the media composition. In-context previewing of secondary media facilitates selection of the optimal secondary material, accelerated workflows, and improved media cost control.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of, under 35 U.S.C.§120, and is a divisional application of pending U.S. application Ser.No. 12/859,942, filed Aug. 20, 2010, which is incorporated herein byreference.

BACKGROUND

When creating a media composition, producers and editors often wish tocombine original media captured specifically for the new compositionwith pre-existing material. Examples of such pre-existing material arestock video footage and stock music. Typically, the pre-existingmaterial, or “secondary material,” is identified and acquired during thepost-production phase of media composition. Current workflows forincorporating such material involve searching catalogs of materialoffered by providers of secondary material, identifying the desiredmaterial, acquiring rights to use the material, including paying for thematerial if required, and receiving a copy of the acquired material,commonly by downloading the material over the Internet. Once thematerial is received, it can be imported into a post production editingsystem. Such a workflow is cumbersome, involving numerous steps.Furthermore, the process is often wasteful, since a proper evaluation ofthe effectiveness of the acquired secondary material is only possibleonce the material has been paid for, downloaded, imported into theediting system, and cut into a sequence. In addition, time-based mediaclips offered by the secondary material providers have a predeterminedduration; an editor may only be interested in using a fragment of aclip, but is required to acquire the entire clip in order to obtain thefragment, with the consequent increase in cost, and download delay.

SUMMARY

Media composition methods and systems feature researching and selectingsecondary footage, previewing it in the context of a media composition,purchasing and downloading it, all from within a media compositionsystem or a media editing system. The in-editor workflow enables rapidin-context preview and editing of secondary materials via streamedproxies that are inserted into media that is already present in themedia composition.

In general in one aspect, a method of creating a time-based mediacomposition, the method including using a computer-based mediacomposition system connected to a wide-area network to search forsecondary material available from secondary material providers connectedto the wide-area network. The method further involves: selecting an itemof secondary material located during the searching step; receiving overthe wide area network descriptive metadata pertaining to the selectedsecondary material item; inserting at a temporal location within thetime-based media composition a representation of the selected secondarymaterial item, wherein the time-based media composition includes primarymedia; and outputting a rendered portion of the time-based mediacomposition spanning the temporal location, the portion including aportion of the primary media and a portion of the secondary materialitem, wherein outputting the secondary material portion involvesoutputting an evaluation version of the secondary material item that isstreamed to the media composition system over the wide area network.

Some embodiments include one or more of the following features. Theselected secondary material item is a video clip or an audio clip.Displaying the outputted rendered portion of the time-based mediacomposition on a display device connected to the media compositionsystem. The selected secondary material item includes at least one of adigitally represented font, a sound effect, a graphical element, a videoeffect and an audio effect. Using the media composition system topurchase at least a portion of the selected secondary material item forinclusion within the time-based media composition. The receiveddescriptive metadata includes a cost associated with usage of thepurchased portion of the secondary material item within the time-basedmedia composition, and the media composition system displays anindication of the cost. Transferring to the media composition systemover the wide-area network an original quality version of the purchasedportion of the secondary material item. Dynamically re-linking theoriginal quality version of the transferred portion of the selectedsecondary material item to the media composition. The receiveddescriptive metadata includes information pertaining to usage rightsavailable for the secondary material item. The inserted location isexpressed as a time offset within the time-based media composition.

Under another aspect, a system for editing a time-based mediacomposition includes a memory for storing computer-readable instructionsand a processor connected to the memory, wherein the processor, whenexecuting the computer-readable instructions, causes the mediacomposition system to: enable a user to search for secondary materialover a wide-area network connected to the media composition system;enable the user to select an item of secondary material identified bythe searching; receive over the wide area network descriptive metadatapertaining to the selected secondary material item; enable the user toinsert at a temporal location within the time-based media composition arepresentation of the selected secondary material item, wherein themedia composition includes primary media; and output a rendered portionof the time-based media composition spanning the temporal location, theportion including a portion of the primary media and a portion of thesecondary material item, wherein outputting the secondary materialportion involves outputting an evaluation version of the secondarymaterial item that is streamed to the media composition system over thewide area network.

Under yet another aspect, a computer program product includes acomputer-readable memory and computer-readable instructions stored inthe computer-readable memory that, when executed by a computer connectedto a wide-area network, instruct the computer to perform a method ofcreating a time-based media composition that includes secondarymaterial, the method including some or all of the features listed above.

Under yet another aspect, a computer-based method for time-based mediacomposition includes: receiving at the computer over a wide area networka stream of data representing a clip of time-based media and descriptivemetadata pertaining to the clip, wherein the clip is encoded in a clipmedia format and wherein the descriptive metadata is encoded in a clipmetadata format; using a media plug-in module to repackage the clip intoa player format for playback by a media player implemented on thecomputer, wherein the player format is not dependent on the clip mediaformat; using a metadata plug-in module to reformat the descriptivemetadata into a player metadata format compatible with the media player,wherein the player metadata format is not dependent on the clip metadataformat; and using the media player to output the repackaged media clipand at least a portion of the descriptive metadata on a displayconnected to the media player.

Various embodiments include one or more of the following features. Theclip of time-based media includes a video clip. The clip of time-basedmedia includes an audio clip. Receiving at the computer over a wide-areanetwork, a stream of data representing a second clip of time-based mediaand descriptive metadata pertaining to the second clip, wherein thesecond clip is encoded in a second clip media format, using a secondmedia plug-in module to repackage the second clip into the playerformat, and using the media player to output the reformatted secondmedia clip on a display connected to the media player. Receiving at thecomputer over a wide-area network, data representing a second clip oftime-based media and descriptive metadata pertaining to the second clip,wherein the descriptive metadata pertaining to the second clip isencoded in a second clip metadata format, using a second metadataplug-in module to reformat the descriptive metadata pertaining to thesecond clip into the player metadata format, and using the media playerto output the reformatted descriptive metadata pertaining to the secondclip.

Under a still further aspect, a method of creating a time-based mediacomposition includes: using a computer-based media composition systemconnected to a wide-area network to search for media available fromsecondary material providers connected to the wide-area network;selecting an item of media located during the searching step; receivingover the wide area network descriptive metadata pertaining to theselected media item; inserting at a temporal location within thetime-based media composition a representation of the selected mediaitem; outputting a rendered portion of the time-based media compositionspanning the temporal location, the portion including a portion of theselected media item, wherein outputting the portion of the selectedmedia item involves outputting an evaluation version of the selectedmedia item that is streamed to the media composition system over thewide area network; and using the media composition system to purchase atleast a portion of the selected media item for inclusion within themedia composition.

Some embodiments feature the following features. The selected mediaitems include a video clip and an audio clip, and a representation ofthe selected audio clip is inserted at a second temporal location withinthe time-based media composition, wherein a time span of the insertedrepresentation of the selected audio clip overlaps with a time span ofthe inserted video clip, and outputting the rendered portion of thetime-based involves playing an evaluation version of the selected audioclip that is streamed to the media composition system over the wide areanetwork.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level block diagram of a media composition system andremote systems involved in incorporating secondary material into a mediacomposition.

FIG. 2 is a flow diagram of a software plug-in for receiving streamingmedia.

FIG. 3 is a flow diagram for incorporating secondary material in a mediacomposition according to the described embodiment.

FIG. 4 is a screenshot of a media composition system illustrating aninterface for composing media with secondary material.

DETAILED DESCRIPTION

The creation of media, especially film, video, and music, often occursin an environment that is constrained by tight financial budgets andtime deadlines. In such an environment, the individuals and teamsinvolved in media creation place a great premium on time efficiency intheir workflows, while at the same time keeping project costs undercontrol. Thus, workflow improvements that reduce the number of stepsrequired during media project creation, reduce lags caused by transferof media over networks, facilitate rapid and effective communicationamong team members, or avoid the need to perform revisions resultingfrom cost overruns are highly advantageous. An aspect of media creationthat commonly introduces inefficiencies, delays, and cost issues is theincorporation of secondary material into the media composition. Themedia composition and editing methods, systems, and computer programproducts described herein address this aspect by providing workflows andfeatures for evaluating and incorporating secondary material into amedia composition.

As used herein, secondary material refers to pre-existing content thatis not created specifically for the media composition being created.Such material generally originates from a source other than the mediacomposition's production team. Secondary material includes, but is notlimited to video clips, audio clips, still imagery, video effects, audioeffects, graphics, and fonts. In the described embodiment, secondarymaterial is stored on a remote secondary material provider systemconnected to a wide area network, such as the Internet. Secondarymaterial is distinguished from primary material, which refers to contentthat is generated by the entity creating the media composition, such asan individual producer or music composer, an editorial team, or anorganization, such as a film studio or a television broadcast company.Primary material is created specifically for the media composition, andmay be stored locally to the media composition system being used tocreate the composition, or on a server connected to the compositionsystem by a high bandwidth network.

Referring to FIG. 1, a system for incorporating secondary materialwithin a media composition includes media composition system 102, whichmay be a non-linear media editing application that runs on a client,such as a computer running Microsoft Windows® or the Mac OS®. Examplesof non-linear media editing applications include Media Composer® fromAvid Technology, Inc. of Burlington, Mass., described in U.S. Pat. Nos.5,267,351 and 5,355,450, which are incorporated by reference herein, andFinal Cut Pro® from Apple Computer, Inc. of Cupertino Calif. Mediacomposition system 102, and optionally one or more additional mediacomposition systems (not shown), are connected to media storage 104 byhigh bandwidth network 106, implemented using such protocols as FibreChannel, InfiniBand, or 10 Gb Ethernet, and supporting bandwidths on theorder of gigabits per second or higher. Media storage 104 may include asingle device or a plurality of devices connected in parallel. Theprimary material being used for the media composition is stored on mediastorage 104, or on storage included within the client system supportingthe non-linear editing application. After acquisition and downloading,secondary material may also be stored on media storage 104.

Media composition system 102 is connected to wide area network 108, suchas the Internet, or the “Cloud” via a low-bandwidth connection 110, suchas an Ethernet network, a wireless network or an Internet connection,having a bandwidth of the order of 1-100 megabits per second or higher.

One or more sources of secondary material 112, 114 are connected to thewide area network, and are accessed by the media composition system viathe low band-width connection and the wide area network. In addition,one or more research stations 116, 118 may also be connected to the widearea network. Research stations may be implemented on a thin clientcomputer.

Secondary material providers offer evaluation materials that enableprospective customers to assess the secondary material. Examples ofevaluation materials include but are not limited to: for video, a lowresolution proxy or a watermarked version; for audio, a low sample rateclip; for still imagery, a low resolution and/or watermarked image; fora video effect, a version of the effect software with a watermark orother feature that indicates an evaluation version is being used; andfor fonts, a subset of the font characters. In order to facilitate theassessment or preview of evaluation materials within the context of amedia composition being edited using media composition system 102, twotypes of plug-in software modules are provided: Media Streaming Plug-in(MSP) 120 and Media Metadata Plug-in (MMP) 122. Together, the MSP andMMP receive and repackage both structural and descriptive metadataassociated with the evaluation materials into a format handled by themedia composition system, such as Advanced Authoring Format (AAF).

MSP 120 handles the structural metadata associated with the evaluationmaterials, enabling streaming media to be browed and edited within thecomposition system without any transcoding, rewrapping, importing,and/or copying of media. In the described embodiment, MSP 120 iscompatible with the plug-in architecture developed by Avid Technology,Inc. for file-based media workflows, and known as Avid® Media Access(AMA). The system may include multiple MSPs, each one handling adifferent media streaming format deployed by the secondary materialproviders. Streaming media formats include but are not limited toQuickTime® and Adobe® Flash®. The evaluation material resides on remotesecondary material sources 112, 114, and may be requested via a HTTPprotocol and streamed in real-time to media composition system 102. Uponreceiving the streamed material, MSP 120 extracts frames from thestreamed packet, decodes it, and sends it to the media compositionsystem, which in turn may output the media.

The high-level actions performed by MSP 120 are illustrated in the flowdiagram shown in FIG. 2. As packets are received from WAN/Cloud 108,each packet from the transport stream is captured (202) and parsed(204). A check is performed to determine if the packet is a video oraudio packet (206), and if not, the packet is ignored and the MSPcaptures the next packet from the transport stream. The packet formatmay be RTP or any of the standardized packet formats for deliveringvideo and audio over the Internet. If the packet is a video or audiopacket, the elementary media stream is extracted (208), and passed on tothe demultiplexer for separating video and audio essence (210), whilethe next packet is captured (202). The media essence is then decoded(212) by a decoder module and the decoded media is output to sampleprovider 214, which provides the supported video and audio formats tothe media composition system. Sample provider 214 also provides amechanism for the media composition system to request video and/or audiosamples in real-time during playback. In response to a request via thesample provider, the MSP provides the desired video frame and/or audiosample, which have been decompressed in decoding step 212. The sampleprovider in turn provides the samples to media composition system 102.

The second type of plug-in module, referred to as Media Metadata Plug-in(MMP) 122 handles the descriptive metadata associated with theevaluation materials, which may include data that originated with themedia itself, or data provided by the secondary material provider.Descriptive metadata includes non-real-time metadata that includesdescriptive information pertaining to the secondary material item as awhole, such as: title; date of production; content summary; names ofindividuals such as a producer, director, composer, actor; price; andscope of available usage rights. Descriptive metadata may also includereal-time metadata that applies to specific sections, frames, orsegments of the secondary material, such as GPS coordinate, scenedescription, closed captions. Each secondary media provider may providedescriptive metadata in one of several standard formats, or in a uniqueproprietary format. The plug-in architecture enables multiple MMPs to beprovided, each one handling the extraction and reformatting of acorresponding provider's metadata to a form that is recognized by themedia composition system. This architecture permits the media playerwithin the media composition system to be abstracted from the sourceformat of the media and its descriptive metadata, with the ability tohandle new remote streaming media and provider metadata formats simplyvia the provision of corresponding new MSP and MMP plug-ins.

Research stations 116, 118 enable media researchers to conduct searchesfor secondary media remotely and independently of an editor and themedia composition system. Each research station runs a search toolapplication provided with versions of the two types of plug-in describedabove in connection with media composition system 102. These plug-ins(124, 126, 128, 130) are adapted to provide metadata in a formrecognized by the search tool application, enabling the researcher tohave full access to all the media and provider metadata.

We now describe an exemplary workflow for incorporating secondarymaterial within a media composition with reference to FIG. 3. In step302, a researcher or an editor, using research station 116, links to oneor more sources of secondary material over a wide area network. In thedescribed embodiment, the research station features a search tool, suchas a browser-based application, running on an internet-connected client.In some embodiments, the researcher is not registered with secondarymaterial providers; in other embodiments, the researcher has apre-established account with one or more secondary material providers,so that after performing a user authentication step (not shown), thesecondary material providers may apply pre-established searchpreferences corresponding to a user profile, such as limiting the searchto a certain format, price range, or territorial licensing regime. Theresearch station may also provide payment details to secondary providersso that paid transactions can occur, for example, when paying to receivea proxy.

After linking to one or more secondary material providers, theresearcher searches for secondary material (step 304). The researchermay limit the search to a single provider, or perform a federated searchacross multiple secondary material providers to obtain aggregated searchresults. For each of the secondary providers selected for searching, theresearcher may use search terms directed to any of the secondarymaterial metadata exposed by the providers. Examples of such metadatainclude, but are not limited to: descriptive metadata, such as a contentsummary (e.g., content description, search terms, geographic location,perspective such as aerial, names of personalities in the clip, eventdescription, film style such as time lapse or effects used, and capturedate), language, and names of individuals or companies responsible forthe content such as the director, producer, composer, musical performer,and actor; structural metadata, such as file format, compression scheme,frame rate, audio sample rate, audio compression; and time-synchronousmetadata, such as camera data, GPS coordinates, subtitles, and closedcaptions.

The researcher then assesses the search results by reviewing theevaluation material offered by the secondary material provider, forexample by listening to a low sample-rate audio clip, or viewing a proxyvideo clip. The researcher selects items of interest that may besuitable for inclusion within the media composition, and places theselected items in a virtual cart (step 306). The cart may be displayedon the research station as a window that includes an indication of theselected items. The indications may include text, thumbnails, andmetadata that identifies and provides information about the selectedmedia. The cart includes a copy of some or all of the metadataassociated with the selected media, optionally a pointer to themetadata, a link to the source of the secondary material, and may alsoinclude a copy of the evaluation material itself.

The secondary media researcher may annotate search results, addingcomments, markers or locators on clips with precision down to anindividual frame. Annotations remain associated with the search resultsand are passed along to the editor at the media composition system whenthe cart is transferred. The editor may add further annotations, andsend these back to the researcher, who may then use the editor'scomments to alter or refine the search. A search result cart may bepassed back and forth between multiple members of a production team,with annotations and approval added at any of the steps along the way.The cart is a small object, and is readily shared over low bandwidthnetworks, such 3G wireless networks that link mobile devices, includingsmart phones and tablet computers.

Metadata provided by secondary media providers may change over time. Forexample, a provider may change pricing, or certain territories availablefor licensing may be added or dropped. As long as a media item isretained in the virtual cart, or is otherwise selected by theresearcher, the secondary material provider may push updated metadata tothe research station. Such updating ensures that media project personnelare always working with up-to-date information.

The research station may include advanced search options to assist theresearcher in satisfying various requirements or constraints of theproject. For example, when the researcher has a budget limit forsecondary material, returned search results can be limited to a selectedcost range (including zero cost for free material), either for eachindividual item, or in aggregate. Another search feature enables theresearcher to select secondary material that satisfies all the searchrequirements except cost, and then request that the system look forlower cost alternatives and/or limit the search to providers offeringadvantageous pricing. Alternatively, the system may automatically selectsearch results not to exceed a pre-determined project budget.

Upon a command from the researcher, or at the request of an editor usingthe media composition system, the contents of the virtual cart aretransferred over the wide area network, and imported into the mediacomposition system (step 308). Referring to FIG. 4, which illustrates ascreen shot of media composition system 102, indications of the importedsearch results are displayed in secondary material bin 402. Any metadatathat is available from the secondary provider is available to the mediacomposition system, and may be displayed in bin 402 to identify anddescribe the media. Furthermore, the metadata is available forgenerating various reports, as described below.

In some embodiments, the virtual cart may be uploaded to anauthenticated server in the Cloud, and accessed upon demand by aneditor. The editor may review the search results on a mobile device,such as a laptop, tablet computer, or smartphone, and upload annotationsand comments for subsequent retrieval by the researcher. Alternatively,the editor accesses the server using the media composition system, andimports the cart directly into a bin within the system.

The editor may browse and edit the imported evaluation versions of thesecondary material directly within the media composition (step 310),such as by dragging an indication of a clip from bin 402 into mediacomposition timeline 404. When working with video or audio secondarymaterial, the secondary media is cut into a sequence, and evaluated inthe context of adjacent clips in the composition's timeline. Referringagain to FIG. 4, the timeline includes both primary media 406 andsecondary media 408, which at this stage represents the evaluationsecondary media. Playing back a sequence of the media composition thatspans both primary media and secondary evaluation media in monitorwindow 410 proceeds without any significant lag when the secondaryevaluation material is played, since the media is streamed directly fromthe remote secondary material provider. Furthermore, transitions, andcertain video or audio effects may be added to the secondary evaluationmaterial. The editor may also embed time-code into the secondary media.For still image secondary material, the editor may add pan and zoomeffects to achieve the desired result. Thus, by making the evaluationversion of the secondary material a “first class citizen” within theediting system, an editor can assess the secondary material within itsintended usage context, avoiding delays and monetary outlays associatedwith prior workflows involving media purchase and download. In addition,metadata pertaining to cost and usage rights may be accessed in realtime by the media composition system, which uses the information togenerate and display costs and distribution rights associated with thepreviewed selection.

Playing of a sequence of the composition involves a player module withinmedia composition system 102, sometimes referred to a “media engine.” Asdescribed above, the player module links to one or more media sources,and optionally to graphics, effects, material for compositing, and otherinformation pertaining to the media composition, and renders it into aform suitable for preview. For a video composition, the rendering takesthe form of a video track and an audio track. In the workflows describedabove, the player outputs the rendered composition to a display forpreviewing by the editor. In other workflows, the output is sent to arecording device for subsequent review, or sent over a network, eitherlocally to another station, or over a wide area network for preview atanother location. The player output may be directed to any device ableto receive the rendered output.

As described above, the media composition system is able to play back apreview of a sequence of the media composition that spans both primarymaterial and secondary evaluation material. For a video composition, toplay back the primary material the system requests and displays framesfrom the primary material stored on primary media storage 104. Invarious embodiments, primary media is stored on shared storage, whichmay be in a remote location accessed over a wide area network. To playback a clip or clip portion of the secondary material evaluationmaterial, the system displays frames from media that is being streamedover the low bandwidth network from the secondary provider. Frames maybe prefetched and buffered in order to ensure a seamless playbacktransition between the primary media and the evaluation secondarymaterial, the extent of the prefetching and buffering being determinedin part by the bandwidth that is available over connection 110 betweenthe media composition system and the wide area network. Prefetching andbuffering may be multi-threaded to help eliminate latencies. Forexample, when playing a sequence including multiple remote secondarymedia clips, the system may spawn a thread for each of the secondarymedia clips, each thread using a separate instance of the required mediaand metadata plug-ins. Multiple threads may also be spawned to prefetchand extract frames from different parts of a single secondary mediaclip. The number of threads spawned is based on the bandwidth availablefor transferring the secondary media to. the media composition systemfrom the remote provider. This scheme helps optimize the networkresources so that an editor can play a cut sequence containing anarbitrary combination of primary, locally stored media, and secondary,remote media from end to end without experiencing any significantbuffering pauses.

Network traffic and any media prefetching delays may also be reduced bycaching the received secondary evaluation material locally to the mediacomposition system. In this scheme, the second and subsequent times aneditor wishes to preview a particular secondary media element, thesystem retrieves the material from the cache rather than requesting itagain from the provider over the Internet.

In order to help distribute network load, and also to facilitate theevaluation of more than one element of secondary material at a time, thesystem may permit a hybrid workflow, in which secondary evaluationmaterials are received at the media composition system both viastreaming (as described above) and by the downloading of proxies. Inthis workflow, the editor may request that proxies for a selection ofsecondary material, such as, for example, all the items in a secondarymaterial bin, be downloaded in the background, while editing andpreviewing of the media composition proceeds. When an item of secondaryevaluation material is required for playback, the system retrieves adownloaded proxy from local storage if it is available, but otherwise,i.e., if the system has not yet been able to download it, the requiredmaterial is streamed in real-time over the network into the mediacomposition system. The MSP and MMP plug-ins reformat the retrievedmedia in a similar fashion, whether it is being remotely streamed, orread from local storage.

An editor may also work in a pure proxy mode, in which all secondaryevaluation media is downloaded to local storage before it is previewedand evaluated within the media composition. In both the hybrid and pureproxy workflows, more than one element of secondary evaluation materialis available to the media composition at a time. For example, the systemmay receive a first proxy by remote streaming, and a second from localstorage. Having more than one proxy available at a time provides theeditor with the ability to preview portions of the composition in whichmore than one item of secondary media is played together, such as in adissolve transition from a first to a second secondary item, or whencompositing multiple secondary clips.

An editor who wishes to preview a secondary audio clip as background toa secondary video clip may cause the system to receive evaluationversions of both the video and audio, insert them both into the mediacomposition in their intended locations, such that they at leastpartially overlap, and preview them together. This provides anin-context means of evaluating the audio in its intended role.

Note that in all of the described workflows, secondary media metadata isavailable to the media composition system at once, prior to mediastreaming or download.

The methods of searching and previewing secondary material describedabove are applicable to the creation and editing of media compositionsfor which no primary media exists. For some such compositions, the finalcomposition includes only secondary material. In others, the secondarymaterial is identified, evaluated and cut into a sequence first, andprimary material is added later. The system permits an editor to previewselected secondary material items in the context of other secondarymaterial items within a composition before making a purchasing decision.As indicated above, several items of secondary material represented bythe provider's evaluation materials may be previewed together, enablingthe secondary media to be evaluated in sequence, optionally withtransitions or compositing involving multiple secondary media clips, andwith certain effects applied. Secondary material that has already beenacquired, downloaded, and linked, as described for subsequent steps inthe workflow below, may be effectively be treated as primary material,even though such material is not original to the media composition.

In sum, the described system provides a realistic “in-context”evaluation environment, greatly assisting editors to make optimal, timeand cost-efficient choices of their secondary material for mediacompositions before purchase.

If the editor decides to incorporate into the media composition some ofthe secondary material corresponding to the evaluation materialsreceived in step 310, the system facilitates transaction step 312.Information may be requested from the editor in order to determine thetype of license required, and the corresponding cost, if any. Theinformation may be stored in a user profile, which may include a list ofpreferred secondary media providers, logins for each provider, paymentaccount details, and a profile of the media composition production. Theproduction profile includes one or more of the project type, e.g.,advertising, corporate/government, documentary, education,entertainment; distribution, e.g., broadcast and cable TV, internet,non-broadcast media; territory; and term. If payment is required, astandard e-commerce transaction is performed.

Licensing and pricing may also be determined in part by the editor'sselection of only a portion of a media clip for inclusion within thecomposition. Thus if a secondary media provider offers reduced pricingfor portions of a time-based media clip, such as pricing pro-rated byduration, the system automatically computes the cost based on theduration of the clip portion that the editor has cut into the mediacomposition sequence. In addition, usage rights may also vary betweenthe various portions of a media clip, and the system may also take thisinto account when preparing the appropriate license. Sub-clip licensingand pricing may offer significant savings for an editor, and also reducemedia transfer times, since only the required portion of original mediais transferred.

Once transaction step 312 is complete, the media composition systemtransfers the acquired original quality media from the secondary mediaprovider to media storage 104 or other local storage over the wide areanetwork and low bandwidth connection 110 (step 314). The transfer ispreferably performed in the background, enabling the editor to continueworking on the composition. After the transfer is completed, the mediacomposition system dynamically relinks the transferred secondarymaterial to the media composition (step 316).

Another embodiment involves a single station workflow, in which thesteps described above in connection with the research stations prior toimporting step 308 are performed by using the media composition system.In this workflow no research station is involved, and the researchphase, as well as the editing and transaction phases are conducted withthe media composition system. In various embodiments, research proceedsin parallel, with one or more media researchers conducting research atlocal or remote research stations, and an editor using the mediacomposition system. In the single station workflow, secondary materialprovider linking step 302, searching and selection of secondary materialstep 304, and placement of an indication of selected secondary materialin a cart or bin step 306 are performed within the media compositionsystem, rather than using an external research station. In thisimplementation, an editor does not need to leave the media compositionapplication while accomplishing all the steps (i.e., 302, 304, 306, 310,312, 314, and 316) required to search, acquire, and incorporatesecondary material into a finished media composition. (Step 308 is notapplicable in this workflow.)

In a variant of the single station workflow, the editor may select asubset of the items in the cart for inclusion within a bin associatedwith a media composition. Alternatively, an editor may circumvent thebin and cart entirely by selecting an item of secondary materialdirectly from a list of search results for immediate preview at aselected location within a media composition. The selected item is addedautomatically to the cart in the background.

We now describe the reporting features enabled by the described system.As described above, during the secondary material research process,remote research stations (116, 118) and/or media composition system 102receive metadata pertaining to the searched and selected secondarymedia. To obtain a current running total of secondary material costs,the system sums the received pricing data, which is displayed to theuser, in textual and/or graphical form, such as bar meter 412 (FIG. 4).Availability of usage rights for various territories may also bedisplayed, either textually or on a map. If secondary material ofinterest includes usage rights that cover only some of the requiredterritories, the system may locate alternative choices, and attempt tominimize the cost by proposing different selections for differentterritories, with each of the selections satisfying the researchcriteria. The reports may be generated on remote research stations aswell as within the media composition system.

Reporting formats may include an annotated bill of materials forsecondary media required to complete a media composition. Anotherreporting technique involves annotating a timeline view of the mediacomposition, in which each element of the timeline includes anannotation as to whether the material is primary or secondary, andoptionally display the cost, the availability of the required usage.licenses, quality, and other parameters of interest to the productionteam.

In order to make the report as accurate as possible, the reportingfeature may exploit any available material information. For example, forcost reporting, cost totals reflect pricing based on factors that mayinclude, but are not limited to the territory in which the media is tobe delivered to a consumer, the nature of the venue or medium in whichthe composition is to be delivered to the consumer, the period of timeover which the media is to be made available to consumers, and the mediaoutput platform. For reports generated within the media compositionsystem, the system has access to additional details of the requiredusage that may affect pricing or usage rights, such as whether an entireclip or only a portion is required, or whether music is used whiletitles are shown in a video composition. In sum, when secondary materialhas been cut into a composition, i.e., when the precise secondarymaterial usage requirements for a deliverable media composition aredetermined, real-time, accurate cost and license reporting becomesavailable.

The various components of the system described herein may be implementedas a computer program using a general-purpose computer system. Such acomputer system typically includes a main unit connected to both anoutput device that displays information to a user and an input devicethat receives input from a user. The main unit generally includes aprocessor connected to a memory system via an interconnection mechanism.The input device and output device also are connected to the processorand memory system via the interconnection mechanism.

One or more output devices may be connected to the computer system.Example output devices include, but are not limited to, liquid crystaldisplays (LCD), plasma displays, cathode ray tubes, video projectionsystems and other video output devices, printers, devices forcommunicating over a low or high bandwidth network, including networkinterface devices, cable modems, and storage devices such as disk ortape. One or more input devices may be connected to the computer system.Example input devices include, but are not limited to, a keyboard,keypad, track ball, mouse, pen and tablet, communication device, anddata input devices. The invention is not limited to the particular inputor output devices used in combination with the computer system or tothose described herein.

The computer system may be a general purpose computer system which isprogrammable using a computer programming language, a scripting languageor even assembly language. The computer system may also be speciallyprogrammed, special purpose hardware. In a general-purpose computersystem, the processor is typically a commercially available processor.The general-purpose computer also typically has an operating system,which controls the execution of other computer programs and providesscheduling, debugging, input/output control, accounting, compilation,storage assignment, data management and memory management, andcommunication control and related services. The computer system may beconnected to a local network and/or to a wide area network, such as theInternet. The connected network may transfer to and from the computersystem program instructions for execution on the computer, media data,metadata, review and approval information for a media composition, mediaannotations, and other data.

A memory system typically includes a computer readable medium. Themedium may be volatile or nonvolatile, writeable or nonwriteable, and/orrewriteable or not rewriteable. A memory system typically stores data inbinary form. Such data may define an application program to be executedby the microprocessor, or information stored on the disk to be processedby the application program. The invention is not limited to a particularmemory system. Time-based media may be stored on and input from magneticor optical discs, which may include an array of local or networkattached discs.

A system such as described herein may be implemented in software orhardware or firmware, or a combination of the three. The variouselements of the system, either individually or in combination may beimplemented as one or more computer program products in which computerprogram instructions are stored on a computer readable medium forexecution by a computer, or transferred to a computer system via aconnected local area or wide are network. Various steps of a process maybe performed by a computer executing such computer program instructions.The computer system may be a multiprocessor computer system or mayinclude multiple computers connected over a computer network. Thecomponents described herein may be separate modules of a computerprogram, or may be separate computer programs, which may be operable onseparate computers. The data produced by these components may be storedin a memory system or transmitted between computer systems.

Having now described an example embodiment, it should be apparent tothose skilled in the art that the foregoing is merely illustrative andnot limiting, having been presented by way of example only. Numerousmodifications and other embodiments are within the scope of one ofordinary skill in the art and are contemplated as falling within thescope of the invention.

1. A computer-based method for time-based media composition, the methodcomprising: receiving at the computer over a wide area network, a streamof data representing a clip of time-based media and descriptive metadatapertaining to the clip, wherein the clip is encoded in a clip mediaformat and wherein the descriptive metadata is encoded in a clipmetadata format; using a media plug-in module to repackage the clip intoa player format for playback by a media player implemented on thecomputer, wherein the player format is not dependent on the clip mediaformat; using a metadata plug-in module to reformat the descriptiveMetadata into a player metadata format compatible with the media player,wherein the player metadata format is not dependent on the clip metadataformat; and using the media player to output the repackaged media clipand at least a portion of the descriptive metadata on a displayconnected to the media player.
 2. The computer-based method of claim 1,wherein the clip of time-based media includes a video clip.
 3. Thecomputer-based method of claim 1, wherein the clip of time-based mediaincludes an audio clip.
 4. The computer-based method of claim 1, furthercomprising: receiving at the computer over a wide-area network, a streamof data representing a second clip of time-based media and descriptivemetadata pertaining to the second clip, wherein the second clip isencoded in a second clip media format; using a second media plug-inmodule to repackage the second clip into the player format; and usingthe media player to output the reformatted second media clip on adisplay connected to the media player.
 5. The computer-based method ofclaim 1, further comprising: receiving at the computer over a wide-areanetwork, data representing a second clip of time-based media anddescriptive metadata pertaining to the second clip, wherein thedescriptive metadata pertaining to the second clip is encoded in asecond clip metadata format; and using a second metadata plug-in moduleto reformat the descriptive metadata pertaining to the second clip intothe player metadata format; and using the media player to output the atleast a portion of the reformatted descriptive metadata pertaining tothe second clip.